Apparatus operation device, apparatus operation method, and electronic apparatus system

ABSTRACT

An apparatus operation device includes a line-of-sight detecting unit that detects a line of sight of a user, a neck-mounted terminal that is mounted around a neck of the user and detects a motion of the neck of the user, a determining unit that determines, based on the line of sight that has been detected and the motion of the neck that has been detected, at least one of an electronic apparatus as a target apparatus to be operated or operation details, and an apparatus control unit that controls the electronic apparatus as the target apparatus in accordance with the determination.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of PCT International Application No.PCT/JP2016/070073 filed on Jul. 7, 2016, which claims priority under 35U.S.C § 119(a) to Japanese Patent Application No. 2015-174575 filed onSep. 4, 2015. Each of the above application(s) is hereby expresslyincorporated by reference, in its entirety, into the presentapplication.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to an apparatus operation device, anapparatus operation method, and an electronic apparatus system, andparticularly relates to a technique for operating an apparatus by usinga line of sight and a motion of the neck of a user.

2. Description of the Related Art

Various types of apparatuses are typically operated by a user withhis/her hand or foot. However, depending on a type or usage situation ofan apparatus, restrictions may be imposed on a user's action or thearrangement of the apparatus or an operation device. For example, in anoperation of a mobile apparatus, such as a smartphone or game machine,driving of a car, or an operation of a medical apparatus, a user may beunable to freely use both hands. As for a medical apparatus, the usermay need to peel off/put on gloves or wash hands before/after operatingthe apparatus. That is, in operation techniques using a hand or foot, itmay be difficult to smoothly perform an operation.

As operation techniques not using a hand or foot, operation techniquesusing a line of sight and a gesture are known (see, for example,JP2014-126997A, JP2000-163196A, JP2001-100903A, and JP2007-195892A).

JP2014-126997A and JP2000-163196A describe that a point of gaze of auser is detected based on a captured image of the head of the user, apredetermined action of the head (for example, a nod action of shakingthe head vertically) is detected based on the captured image of the headof the user, and an operation corresponding to the point of gaze of theuser is decided on upon the predetermined action of the head beingdetected.

JP2001-100903A describes that an action of a hand of a user is detectedbased on a captured image of the hand of the user, and a process ofmoving a cursor to a line-of-sight position is started (turn on a cursorfollowing mode) if one finger is raised, whereas the process of movingthe cursor to the line-of-sight position is finished (turn off thecursor following mode) if no finger is raised but a first is made.

JP2007-195892A describes that a line-of-sight position is calculatedbased on an image captured by a camera of a unit that can be mounted onthe head of a user (line-of-sight position input unit) and a focusposition is moved to the line-of-sight position. JP2007-195892A alsodescribes that a magnetic sensor for detecting a position and an angleof the head of the user is used.

In addition, an operation technique using a voice input is known.JP2001-299691A describes that, if a user presses a foot switch andinputs a name (for example, “decrease light intensity”) of an icon (forexample, a “decrease light intensity” icon) by voice while gazing at theicon displayed on a display unit for a predetermined period of time ormore, the operation of the user is decided on and an actioncorresponding to the icon is performed. The voice input is performed byusing a pin microphone that is attached to clothes or a head microphonethat is mounted on the head.

SUMMARY OF THE INVENTION

However, there has not been provided an operation technique that enablesa user to easily perform an operation even if both hands are occupiedand that enables an operation intended by the user to be appropriatelydetected and performed.

In the techniques described in JP2014-126997A and JP2000-163196A, anoperation determined based on detection of a line of sight is decided onupon a predetermined action of the head of a user being detected, andthus the effect of avoiding a malfunction can be more expected than inthe case of only performing determination based on detection of a lineof sight. However, since an action of the head of the user is recognizedbased on an image that is obtained by photographing the head, it isdifficult for the apparatus to appropriately determine whether themotion of the head results from a motion of neck muscles according tothe user's intention or results from an unconscious motion of the body.If it is assumed that a determination process is modified to adetermination process in which a small motion or a simple motionperformed by a user is ignored by the apparatus and in which only alarge motion (for example, a motion of tilting the head by 45 degrees ormore) or a complicated motion performed by the user is regarded aseffective, the user is forced to perform a large motion or a complicatedmotion. Accordingly, it is impossible to perform a smooth operationalthough a malfunction can be avoided. In addition, the user will gettired.

Also in the techniques described in JP2001-100903A, JP2007-195892A, andJP2001-299691A, it is not possible to achieve both an easy operationunder a situation where both hands of a user are occupied andappropriate detection and execution of an operation intended by theuser.

On the other hand, under circumstances where there is an issue ofmedical errors, recording of conversation between medical doctors may beconsidered to determine the propriety of surgery. However, music isplayed during surgery in a surgery room and a plurality of people isinvolved in the surgery. Thus, in the voice input according toJP2001-299691, ambient music and voice are recorded, and it is notpossible to accurately record the situation of the surgery.

The present invention has been made in view of the above-describedcircumstances, and an object of the present invention is to provide anapparatus operation device, an apparatus operation method, and anelectronic apparatus system that enable a user to easily perform anoperation even if both hands are occupied and that enable an operationintended by the user to be appropriately detected and performed.

To achieve the above-described object, an apparatus operation deviceaccording to a first aspect of the present invention includes aline-of-sight detecting unit that detects a line of sight of a user; aneck-mounted terminal that is mounted around a neck of the user anddetects a motion of the neck of the user; a determining unit thatdetermines, based on the line of sight that has been detected and themotion of the neck that has been detected, at least one of a targetapparatus to be operated or operation details for the target apparatus;and a control unit that controls the target apparatus in accordance withthe determination.

According to this aspect, an operation can be easily performed even ifboth hands of a user are occupied, and an operation intended by the usercan be appropriately detected and performed.

In an apparatus operation device according to a second aspect of thepresent invention, the determining unit determines, based on the line ofsight that has been detected, at least one of the target apparatus orthe operation details, and confirms the determination, based on themotion of the neck that has been detected.

In an apparatus operation device according to a third aspect of thepresent invention, the line-of-sight detecting unit detects a movementof the line of sight, and the determining unit determines, based on themovement of the line of sight that has been detected, at least one ofthe target apparatus or the operation details.

In an apparatus operation device according to a fourth aspect of thepresent invention, the operation details include a function that thecontrol unit causes the target apparatus to execute, and an executioncondition for the function.

In an apparatus operation device according to a fifth aspect of thepresent invention, the apparatus operation device includes a voicerecognizing unit that recognizes a voice of the user by using avibration of a throat of the user, the vibration being detected by theneck-mounted terminal.

In an apparatus operation device according to a sixth aspect of thepresent invention, the apparatus operation device includes a voicerecording unit that records the voice that has been recognized.

In an apparatus operation device according to a seventh aspect of thepresent invention, the line-of-sight detecting unit suspends detectionof the line of sight upon detection of the motion of the neck beingstarted by the neck-mounted terminal, and starts the detection of theline of sight upon the detection of the motion of the neck beingfinished by the neck-mounted terminal.

In an apparatus operation device according to an eighth aspect of thepresent invention, the determining unit invalidates detection of theline of sight if the motion of the neck that has been detected is largerthan a threshold.

To achieve the above-described object, an apparatus operation methodaccording to the present invention is an apparatus operation method foran apparatus operation device including a line-of-sight detecting unitthat detects a line of sight of a user and a neck-mounted terminal thatis mounted around a neck of the user and detects a motion of the neck ofthe user, the apparatus operation method including a determination stepof determining, based on the line of sight that has been detected andthe motion of the neck that has been detected, at least one of a targetapparatus to be operated or operation details for the target apparatus;and a control step of controlling the target apparatus in accordancewith the determination.

To achieve the above-described object, an electronic apparatus systemaccording to the present invention includes the apparatus operationdevice and an electronic apparatus which is the target apparatus.

The electronic apparatus is a medical apparatus, for example.

According to the present invention, an operation can be easily performedeven if both hands of a user are occupied, and an operation intended bythe user can be appropriately detected and performed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating the overall configuration of an exampleof an electronic apparatus system according to a first embodiment;

FIG. 2 is a block diagram illustrating an example of an apparatusoperation device according to the first embodiment;

FIG. 3 is a diagram illustrating an example of screen information fordesignating a target apparatus to be operated;

FIGS. 4A and 4B are diagrams illustrating examples of screen informationfor designating operation details;

FIGS. 5A to 5C are explanatory diagrams for describing detection of amotion of a neck;

FIG. 6 is a flowchart illustrating a flow of a process of a firstexample of an apparatus operation method using the apparatus operationdevice according to the first embodiment;

FIG. 7 is an explanatory diagram for describing detection of a line ofsight in the apparatus operation device according to the firstembodiment;

FIG. 8 is a flowchart illustrating a flow of a process of a secondexample of the apparatus operation method using the apparatus operationdevice according to the first embodiment;

FIG. 9 is a flowchart illustrating a flow of a process of a thirdexample of the apparatus operation method using the apparatus operationdevice according to the first embodiment;

FIG. 10 is a block diagram illustrating an example of an apparatusoperation device according to a second embodiment;

FIG. 11 is an explanatory diagram for describing a visual field imageand images of eyes that are captured by a head-mounted terminal in theapparatus operation device according to the second embodiment;

FIGS. 12A to 12C are explanatory diagrams for describing an example ofdetection of a line of sight and determination of an operation in theapparatus operation device according to the second embodiment;

FIG. 13 is a diagram illustrating the overall configuration of anexample of an electronic apparatus system in a case where an endoscopeapparatus is used as an electronic apparatus;

FIG. 14 is a block diagram illustrating the configuration of theelectronic apparatus system illustrated in FIG. 13; and

FIG. 15 is a diagram illustrating a state of observation using theendoscope apparatus.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, embodiments of an apparatus operation device, an apparatusoperation method, and an electronic apparatus system according to thepresent invention will be described with reference to the attacheddrawings.

First Embodiment

FIG. 1 is a diagram illustrating the overall configuration of an exampleof an electronic apparatus system according to a first embodiment of thepresent invention, and FIG. 2 is a block diagram illustrating an exampleof an apparatus operation device according to the first embodiment ofthe present invention.

As illustrated in FIG. 1, an electronic apparatus system 10 includeselectronic apparatuses 20, each of which is a target apparatus to beoperated, and an apparatus operation device 30 that controls theelectronic apparatuses 20.

The apparatus operation device 30 includes a display unit 31 (alsoreferred to as a “display”) that performs display for a user, aline-of-sight detecting camera 32 that captures an image of the face ofthe user to detect a line of sight of the user, a neck-mounted terminal33 that is mounted around the neck of the user and detects a motion ofthe neck of the user, and an arithmetic unit 34 that serves as a mainbody unit, which controls the display unit 31, the line-of-sightdetecting camera 32, and the neck-mounted terminal 33, and that performsvarious arithmetic operations.

The display unit 31 is constituted by a display device, such as a liquidcrystal display device.

The line-of-sight detecting camera 32 is located near the display unit31, captures an image of at least an eye of the user to generate animage (hereinafter referred to as a “user image”) including the image ofthe eye, and outputs the generated image. In the case of extracting onlythe portion of an eye in the face of the user, the extraction may beperformed by using a function of the line-of-sight detecting camera 32,or a function of the extraction may be provided in the arithmetic unit34. To detect a line of sight reliably and stably, a plurality ofline-of-sight detecting cameras 32 may be provided.

The neck-mounted terminal 33 is a terminal that is to be mounted aroundthe neck of the user and includes, as illustrated in FIG. 1, a neckmotion sensor 52 that detects a motion of the neck of the user and avibration sensor 54 that detects a vibration of the throat of the useron the neck.

As the neck motion sensor 52, a sensor device, such as a myoelectricsensor that detects a motion of a neck muscle, a distortion sensor, oran acceleration sensor, is used. A sensor device other than thesesensors may be used. As “a motion of the neck”, any of a motion of aneck muscle, a motion of a neck bone, or a motion of a neck surface(i.e., a motion of a neck skin) may be detected by the neck motionsensor 52. A plurality of types of sensor devices may be used to performdetection.

As the vibration sensor 54, an acceleration sensor that detects avibration of a neck bone is used, for example. Another type of sensormay be used.

A single sensor device may be implemented as the neck motion sensor 52and the vibration sensor 54, and the single sensor device may be causedto detect a motion of the neck and a vibration of the throat. Forexample, a single acceleration sensor is implemented as the neck motionsensor 52 and the vibration sensor 54, a low-frequency component isextracted from an output signal of the acceleration sensor to generate asignal representing a motion of the neck, and a high-frequency componentis extracted from the output signal to generate a signal representing avibration of the throat.

As illustrated in FIG. 2, the arithmetic unit 34 includes acommunication unit 38 that communicates with the plurality of electronicapparatuses 20, a storage unit 39 that stores a program for controllingthe operation of the electronic apparatuses 20 and information that isnecessary to execute the program, and a central processing unit (CPU) 40that controls the individual units of the apparatus operation device 30by executing the program stored in the storage unit 39.

The communication unit 38 is constituted by a communication device thatcommunicates with the electronic apparatuses 20. FIG. 1 illustrates acase where the communication unit 38 communicates with the electronicapparatuses 20 in a wireless manner, but the communication unit 38 maycommunicate with the electronic apparatuses 20 in a wired manner througha local area network or the like. In addition, the communication unit 38may communicate with a device outside the arithmetic unit 34 (forexample, a database 22), which is not illustrated in FIG. 1, through alocal area network or the like.

The storage unit 39 includes a nonvolatile storage device that storesprograms and the like and a readable/writable storage device that isused as a working area of the programs.

The CPU 40 includes, as illustrated in FIG. 2, a display control unit 41that performs display control for the display unit 31; a line-of-sightdetecting unit 42 that detects, based on a user image output from theline-of-sight detecting camera 32, a line of sight of a user; an actionrecognizing unit 43 that recognizes, based on a motion of the neck ofthe user that has been detected by the neck motion sensor 52 of theneck-mounted terminal 33, an action of the head of the user; adetermining unit 44 that determines, based on the line of sight detectedby the line-of-sight detecting unit 42 and the action of the head of theuser that has been recognized based on the motion of the neck of theuser, at least one of a target apparatus to be operated or operationdetails for the target apparatus; a voice recognizing unit 45 thatrecognizes a voice of the user by using a vibration of the throat of theuser detected at the neck of the user by the vibration sensor 54 of theneck-mounted terminal 33; and an apparatus control unit 46 that controlsthe electronic apparatuses 20 in accordance with the determination madeby the determining unit 44.

The display control unit 41 has a function of causing the display unit31 to display screen information that is necessary for a user operation.

As illustrated in FIG. 3, the display control unit 41 causes the displayunit 31 to display screen information (hereinafter referred to as“target apparatus designation screen information”) that allows a user todesignate, by a line-of-sight input, an icon (for example, an icon E11)representing a target apparatus to be operated (for example, “electronicapparatus A”) among a plurality of icons E11 and E12 respectivelyrepresenting the plurality of electronic apparatuses 20 (for example,“electronic apparatus A” and “electronic apparatus B”).

Also, as illustrated in FIG. 4A, the display control unit 41 causes thedisplay unit 31 to display screen information (hereinafter referred toas “function designation screen information”) that allows a user todesignate, by a line-of-sight input, an icon (for example, an icon E21)representing a function to be executed by a target apparatus to beoperated (for example, “function 1”) among a plurality of icons E21,E22, and E23 respectively representing a plurality of functions (forexample, “function 1”, “function 2”, and “function 3”).

Also, as illustrated in FIG. 4B, the display control unit 41 causes thedisplay unit 31 to display screen information (hereinafter referred toas “execution condition designation screen information”) that allows auser to designate, by a line-of-sight input, an icon (for example, anicon E31) representing an execution condition for a function to beexecuted by a target apparatus to be operated among a plurality of iconsE31, E32, and E33 respectively representing a plurality of executionconditions (for example, “execution condition 1”, “execution condition2”, and “execution condition 3”).

“Function designation screen information” and “execution conditiondesignation screen information” are examples of screen information thatallows a user to designate, by a line-of-sight input, icons representingoperation details (hereinafter referred to as “operation detailsdesignation screen information”). That is, “operation details” in thisexample include “function” and “execution condition”.

Next, line-of-sight detection performed by the line-of-sight detectingunit 42 will be described.

A line of sight (point of gaze) can be detected from an image of an eyeof the user, based on the position of the iris, which is a movable point(a portion that moves relative to a reference point), relative to theinner corner of the eye, which is a reference point. For example, in acase where the iris of the left eye is far from the inner corner of theleft eye, the user is looking left. On the other hand, in a case wherethe iris of the left eye is close to the inner corner of the left eye,the user is looking right.

A line of sight can also be detected based on the position of the pupil,which is a movable point, relative to the position of the cornealreflection, which is a reference point. For example, in a case where thepupil of the left eye is closer than the corneal reflection to the outercorner side of the left eye, the user is looking left. In a case wherethe pupil of the left eye is closer than the corneal reflection to theinner corner side of the left eye, the user is looking right. In thecase of using this method, the face of the user is irradiated withinfrared rays and photographed by an infrared camera.

The line-of-sight detecting unit 42 of this example detects a movementof a line of sight (i.e., continuously detects a line of sight), and thedetermining unit 44 of this example determines, based on the movement ofthe line of sight that has been detected, a target apparatus to beoperated and operation details for the target apparatus. The determiningunit 44 may determine, based on the movement of the line of sight thathas been detected, at least one of a target apparatus to be operated oroperation details.

Next, a specific example of action recognition by the action recognizingunit 43 will be described.

Examples of an action that is recognized by the action recognizing unit43 include “head tilt” (an action of tilting the head) illustrated inFIG. 5A, “nod” (an action of shaking the head vertically) illustrated inFIG. 5B, and “head shake” (an action of shaking the head horizontally)illustrated in FIG. 5C. That is, in response to an instruction from thebrain of the user who intends to perform an action, such as “head tilt”,“nod”, or “head shake”, muscles of the neck are moved and accordinglythe head is moved. Action recognition is performed by directly detecting“a motion of the neck” by the neck motion sensor 52 of the neck-mountedterminal 33, not by detecting “a motion of the head”, which is a resultof a motion of the neck. Therefore, the action recognizing unit 43 iscapable of appropriately recognizing an action intended by the user.

The determining unit 44 determines, based on the line of sight detectedby the line-of-sight detecting unit 42, at least one of a targetapparatus to be operated or operation details, and confirms thedetermination, based on the motion of the neck detected by theneck-motion sensor 52 of the neck-mounted terminal 33. The determiningunit 44 of this example makes a determination based on an actionrecognized by the action recognizing unit 43 by using a detection resultobtained by the neck motion sensor 52. Such a case is also included inthe concept of “to make a determination based on a motion of the neck”or “to confirm the determination based on a motion of the neck” in thepresent invention.

The voice recognizing unit 45 recognizes a voice of the user, based on avibration of the throat of the user detected at the neck of the user bythe vibration sensor 54 of the neck-mounted terminal 33.

The apparatus control unit 46 is an example of a “control unit”according to the present invention and controls, in accordance with adetermination made by the determining unit 44, an electronic apparatus20 which is a target apparatus to be operated.

For example, if the determining unit 44 determines that “electronicapparatus A” has been designated from the target apparatus designationscreen information illustrated in FIG. 3, determines that “function 1”has been designated from the function designation screen informationillustrated in FIG. 4A, and determines that “execution condition 1” hasbeen designated from the execution condition designation screeninformation illustrated in FIG. 4B, the communication unit 38 transmitsan instruction to execute function 1 under execution condition 1 toelectronic apparatus A which is a target apparatus to be operated.

FIG. 6 is a flowchart illustrating a flow of a process of a firstexample of an apparatus operation method using the apparatus operationdevice 30 of this embodiment. The process is executed by the CPU 40 inaccordance with a program stored in advance in the storage unit 39.

Upon the process being started, in step S102, the display control unit41 causes the display unit 31 to display the target apparatusdesignation screen information, as illustrated in FIG. 3.

Subsequently, the process proceeds to step S104, where the line-of-sightdetecting unit 42 detects a line of sight of a user. For example, if theline of sight of the user is directed on the screen of the display unit31, as illustrated in FIG. 7, a position at which the line of sight ofthe user is directed (a line-of-sight position) on the screen of thedisplay unit 31 is calculated based on an image (user image) captured bythe line-of-sight detecting camera 32.

Subsequently, the process proceeds to step S108, where the determiningunit 44 determines, based on the line of sight detected in step S104, atarget apparatus to be operated that has been designated by the user byusing the line of sight.

Subsequently, the process proceeds to step S112, where the neck motionsensor 52 of the neck-mounted terminal 33 detects a motion of the neckof the user and the action recognizing unit 43 recognizes an action ofthe user.

Subsequently, the process proceeds to step S114, where the determiningunit 44 determines whether or not the user has performed an action of aconfirmation operation (for example, “nod” illustrated in FIG. 5B).

If it is determined that a confirmation operation has not been performed(NO in step S114), the process proceeds to step S116, where thedetermining unit 44 determines whether or not the user has performed anaction of a cancellation operation (for example, “head shake”illustrated in FIG. 5C). If it is determined that a cancellationoperation has been performed (YES in step S116), the determinationresult about the target apparatus to be operated that is based on theline-of-sight detection result is cleared, and the process returns tostep S102. If it is determined that a cancellation operation has notbeen performed (NO in step S116), the process returns to step S112,where detection of a motion of the neck (step S112) and determination ofa confirmation operation (step S114) are repeated.

If it is determined that a confirmation operation has been performed(YES in step S114), the process proceeds to step S122, where the displaycontrol unit 41 causes the display unit 31 to display the operationdetails designation screen information, as illustrated in FIGS. 4A and4B. For example, the “function designation screen information”illustrated in FIG. 4A, the “execution condition designation screeninformation” illustrated in FIG. 4B, or screen information includingboth the pieces of screen information is displayed as the operationdetails designation screen information.

Subsequently, the process proceeds to step S124, where the line-of-sightdetecting unit 42 detects a line of sight of the user. That is, theline-of-sight detecting unit 42 calculates, based on an image (userimage) captured by the line-of-sight detecting camera 32, a position atwhich the line of sight of the user is directed (line-of-sight position)on the screen of the display unit 31.

Subsequently, the process proceeds to step S128, where the determiningunit 44 determines, based on the line of sight detected in step S124,operation details designated by the user by using the line of sight.That is, the determining unit 44 determines operation details for theelectronic apparatus 20 as a target to be operated.

Subsequently, the process proceeds to step S132, where the neck motionsensor 52 of the neck-mounted terminal 33 detects a motion of the neckof the user and the action recognizing unit 43 recognizes an action ofthe user.

Subsequently, the process proceeds to step S134, where the determiningunit 44 determines whether or not the user has performed an action of aconfirmation operation (for example, “nod” illustrated in FIG. 5B).

If it is determined that a confirmation operation has not been performed(NO in step S134), the process proceeds to step S136, where thedetermining unit 44 determines whether or not the user has performed anaction of a cancellation operation (for example, “head shake”illustrated in FIG. 5C). If it is determined that a cancellationoperation has been performed (YES in step S136), the determinationresult about the operation details that is based on the line-of-sightdetection result is cleared, and the process returns to step S122. If itis determined that a cancellation operation has not been performed (NOin step S136), the process returns to step S132, where detection of amotion of the neck (step S132) and determination of a confirmationoperation (step S134) are repeated.

If it is determined that a confirmation operation has been performed(YES in step S134), the process proceeds to step S138, where thedesignated function of the designated target apparatus to be operated isexecuted in accordance with the designated operation details.

Subsequently, the process proceeds to step S142, where it is determinedwhether or not to continue an operation of the same electronic apparatus20 among the plurality of electronic apparatuses 20. In the case ofcontinuing an operation of the same electronic apparatus 20 (YES in stepS142), the process returns to step S122. In step S144, it is determinedwhether or not to start an operation of another electronic apparatus 20.In the case of starting an operation of another electronic apparatus 20(YES in step S144), the process returns to step S102.

In the case of finishing the operation of the same and anotherelectronic apparatus 20 (NO in step S144), the process proceeds to stepS150, where voice processing is executed to generate and output areport.

The details of the voice processing in step S150 will be described.First, the vibration sensor 54 of the neck-mounted terminal 33 detects avibration of the throat at the neck of the user. Subsequently, the voicerecognizing unit 45 recognizes, based on the detected vibration of thethroat of the user, a voice of the user. Subsequently, the recognizedvoice is recorded. That is, only the voice of the person who is wearingthe neck-mounted terminal 33 can be easily recorded, without ambientmusic or sound being recorded.

Modes of recording a voice include a mode of performing recording insidethe arithmetic unit 34, which is a main body unit of the apparatusoperation device 30 (hereinafter referred to as an “inside recordingmode”), and a mode of performing recording outside the arithmetic unit34, which is a main body unit of the apparatus operation device 30(hereinafter referred to as an “outside recording mode”), either ofwhich may be used in the present invention. In the inside recordingmode, voice data is recorded in, for example, the storage unit 39 undercontrol of the CPU 40. In this example, the storage unit 39 is anexample of a “voice recording unit”. In the outside recording mode,voice data is output from the communication unit 38 (an example of a“voice data output unit”) and is recorded in, for example, the database22. In this example, the database 22 is an example of the “voicerecording unit”. For example, the voice data is written on a file and istransmitted to the database 22 through a local area network.

Although voice processing (step S150) is performed after the operationof the apparatus has been finished in FIG. 6, the present invention isnot limited to such a case. Voices during the operation of the apparatuscan be input, recognized, and recorded. For example, instructions andconversations of medical doctors and so forth during surgery can berecorded as voice data, which can be used as evidence if a medicalaccident occurs or can be used to pass on medical technologies.

After a report has been generated and output in the voice processing(step S150), the process ends.

In FIG. 6, steps S108, S114, S116, S128, S134, and S136 are an exampleof a “determination step” in the present invention. Step S138 is anexample of a “control step” in the present invention.

In FIG. 6, line-of-sight detection by the line-of-sight detecting unit42 and motion-of-neck detection by the neck-mounted terminal 33 areperformed at different timings, but the line-of-sight detection and themotion-of-neck detection may be performed chronologically in parallel.In addition, each of the line-of-sight detection and the motion-of-neckdetection may be performed continuously. In a case where such parallelcontrol is performed, there is a possibility that the line-of-sightdetection and the motion-of-neck detection are simultaneously performed.However, the line-of-sight detecting unit 42 in this example suspendsdetection of a line of sight upon detection of a motion of the neckbeing started by the neck-mounted terminal 33, and starts detection of aline of sight upon detection of a motion of the neck being finished bythe neck-mounted terminal 33. For example, the line-of-sight detectingunit 42 suspends detection of a line of sight when detection of a motionof the neck is started in step S112 or step S132 in FIG. 6, and restartsdetection of a line of sight when detection of a motion of the neck isfinished.

FIG. 8 is a flowchart illustrating a flow of a process of a secondexample of the apparatus operation method using the apparatus operationdevice 30 of this embodiment. This process is executed by the CPU 40 inaccordance with a program stored in advance in the storage unit 39. Thesame steps as those in the first example of this embodiment illustratedin FIG. 6 are denoted by the same numerals, and only different pointswill be described below.

In this example, line-of-sight detection and motion-of-neck detection(step S104A and step S124A) are performed instead of the line-of-sightdetection (step S104 and step S124) illustrated in FIG. 6. After theline-of-sight detection and the motion-of-neck detection (step S104A andstep S124A), the motion of the neck detected by the neck-mountedterminal 33 is compared with a threshold (step S106 and step S126). Ifthe detected motion of the neck is larger than the threshold, theline-of-sight detection is invalidated.

That is, in this example, line-of-sight detection in a state where theneck of the user is stationary or slightly moving is validated, whereasline-of-sight detection in a state where the neck of the user is largelymoving is invalidated. This is because, in a state where the neck of theuser is largely moving, the head of the user is also largely moving, andas a result there is a possibility that the line of sight of the userwill largely move in an unconscious way and that a determination notintended by the user will be made.

Here, examples of “a case where the detected motion of the neck islarger than the threshold” include a case where the amount of thedetected motion of the neck (i.e., the magnitude of a movement vector)is larger than the threshold, a case where an evaluation valuerepresenting a fluctuation of the direction of the detected motion ofthe neck (i.e., the direction of a movement vector) is larger than thethreshold, and so forth. That is, examples of the magnitude of themotion of the neck of the user that affects the accuracy ofline-of-sight detection include the magnitude of fluctuation of thedirection of a movement vector, as well as the magnitude of the movementvector. In other words, the “detected motion of the neck” conceptuallyincludes an evaluation value of a movement vector as well as themagnitude of the movement vector.

FIG. 9 is a flowchart illustrating a flow of a process of a thirdexample of the apparatus operation method using the apparatus operationdevice 30 of this embodiment. This process is executed by the CPU 40 inaccordance with a program stored in advance in the storage unit 39. Thesame steps as those in the first example of this embodiment illustratedin FIG. 6 and those in the second example of this embodiment illustratedin FIG. 8 are denoted by the same numerals, and only different pointswill be described below.

In this example, line-of-sight detection and motion-of-neck detection(step S112A and step S132A) are performed instead of the motion-of-neckdetection (step S112 and step S132) illustrated in FIGS. 6 and 8. Basedon the results of the line-of-sight detection and the motion-of-neckdetection (step S112A and step S132A), it is determined whether or not aconfirmation operation has been performed (step S114 and step S134) andwhether or not a cancellation operation has been performed (step S116and step S136).

That is, in this example, it is determined based on not only a result ofmotion-of-neck detection but also a result of line-of-sight detectionwhether or not to confirm the determination of a target apparatus to beoperated and operation details and whether or not to cancel thedetermination of the target apparatus to be performed and the operationdetails. This is because, if a determination of whether or not the userhas performed a confirmation operation or a cancellation operation ismade based on only a motion of the neck, a wrong determination that theuser has performed a confirmation operation or a cancellation operationmay be made if the user moves the neck in an unconscious way. As amethod for avoiding such a wrong determination, a method for changing adetermination criterion value, such as increasing a threshold to becompared with a detected motion of the neck, may be used. However, ifthe user is forced to make a larger motion of the neck, that is, to makea forced action, to avoid a wrong determination, the user is unable toperform a smooth operation, which may make the user tired. Thus, adetermination is made by using a characteristic that, if the user movesthe neck while gazing at a graphical user interface (GUI) element (forexample, a button) on the screen of the display unit 31, a line-of-sightposition on the screen of the display unit 31 is substantially constantor within a range of slight movements regardless of a motion of theneck.

For example, in the case of determining whether or not the user hasperformed a confirmation operation by determining whether or not theuser has performed a vertical head shake (nod), when the user shakes thehead downward (or upward), the user's pupils move upward (or downward),so that it is detected that the magnitude of a movement vector of theneck becomes larger than a threshold for determining a motion of theneck, whereas it is detected that an amount of movement of aline-of-sight position on the screen becomes smaller than or equal to athreshold for determining a line-of-sight movement. In the case ofdetermining whether or not the user has performed a cancellationoperation by determining whether or not the user has performed ahorizontal head shake, when the user shakes the head leftward (orrightward), the user's pupils move rightward (or leftward), so that itis detected that the magnitude of a movement vector of the neck becomeslarger than a threshold for determining a motion of the neck, whereas itis detected that an amount of movement of a line-of-sight position onthe screen becomes smaller than or equal to a threshold for determininga line-of-sight movement.

Second Embodiment

FIG. 10 is a block diagram illustrating an example of an apparatusoperation device 30 according to a second embodiment. In FIG. 10, thesame elements as those in the apparatus operation device 30 of the firstembodiment illustrated in FIG. 2 are denoted by the same numerals, andonly different points will be described below.

A head-mounted terminal 60 is a terminal that is to be mounted on thehead of a user, as illustrated in FIG. 11, and includes a visual fieldcamera 61 that captures an image of a photographic subject in a visualfield of an eye of the user and obtains a visual field image, and aline-of-sight detecting camera 62 that captures an image of an eye ofthe user and obtains the image of the eye. The visual field camera 61 islocated so as to be positioned near the left eye and/or the right eyewhen the head-mounted terminal 60 is mounted on the head of the user. Inthis example, visual field cameras 61 are located near the left eye andnear the right eye, respectively, to obtain a stereo image that is madeup of a visual field image of the left eye and a visual field image ofthe right eye, so that a relative three-dimensional positionalrelationship between each of the user's left eye and right eye and thedisplay unit 31 can be calculated. The line-of-sight detecting camera 62is located so that each of the left eye and the right eye becomes animaging target when the head-mounted terminal 60 is mounted on the headof the user. A plurality of line-of-sight detecting cameras 62 (forexample, two line-of-sight detecting cameras: a line-of-sight detectingcamera for capturing an image of the left eye and a line-of-sightdetecting camera for capturing an image of the right eye) may beprovided to detect a line of sight reliably and stably.

An apparatus operation method using the apparatus operation device 30 ofthis embodiment can be executed almost similarly to the processdescribed above by using FIG. 6 in the first embodiment. In the secondembodiment, in line-of-sight detection by the line-of-sight detectingunit 42 (step S104 in FIG. 6), a line-of-sight position of the user iscalculated based on visual field images obtained by the visual fieldcameras 61 of the head-mounted terminal 60 and images of the eyesobtained by the line-of-sight detecting cameras 62 of the head-mountedterminal 60.

The head-mounted terminal 60 of this example continues to transmitvisual field images obtained by the visual field cameras 61 and imagesof the eyes obtained by the line-of-sight detecting cameras 62 to thearithmetic unit 34, which is a main body unit. For example, thehead-mounted terminal 60 transmits the visual filed images and theimages of the eyes as still images at a regular or irregular timeinterval. The head-mounted terminal 60 may transmit the visual filedimages and the images of the eyes as motion pictures at a regular framerate.

The line-of-sight detecting unit 42 of the arithmetic unit 34calculates, based on visual field images (in this example, both a visualfield image of the left eye and a visual field image of the right eye),a relative positional relationship between the eyes of the user and thescreen of the display unit 31. In addition, the line-of-sight detectingunit 42 of the arithmetic unit 34 calculates, based on images of theeyes (in this example, both an image of the left eye and an image of theright eye), a relative positional relationship between a reference pointand a movable point of the eyes (a portion that moves relative to thereference point). In a case where the line-of-sight detecting camera 62captures an image by using visible light, for example, the inner cornerand/or the outer corner of an eye is regarded as a reference point andthe iris is regarded as a movable point. In a case where theline-of-sight detecting camera 62 captures an image by using infraredlight, for example, the corneal reflection point is regarded as areference point and the pupil is regarded as a movable point.

Subsequently, the line-of-sight detecting unit 42 of the arithmetic unit34 calculates, based on the calculated relative positional relationshipbetween the eyes and the screen of the display unit 31 and thecalculated positional relationship between the reference points and themovable points of the eyes, coordinates indicating a position at whichthe user is looking in a visual field image (a line-of-sight position inthe visual field image). That is, the visual field image and theline-of-sight position are associated with each other.

As illustrated in FIG. 12A, in a case where the horizontal direction ofa visual field image is an X direction and the vertical direction is a Ydirection, a line-of-sight position is represented as a range of a smallcircle having a certain radius and centered around the (x, y)coordinates. That is, as illustrated in FIG. 12B, it can be determinedthat the line of sight of the user is directed at a GUI element existingat a portion centered around the (x, y) coordinates in the visual fieldimage (in the figure, the element E21 among three elements: the elementsE21 to E23). As illustrated in FIG. 12C, if the user shakes the headvertically while gazing at the GUI element, a motion of the neck isdetected by the neck motion sensor 52 of the neck-mounted terminal 33and a vertical shake of the head (also referred to as “nod”) of the useris recognized by the action recognizing unit 43. Accordingly, anoperation corresponding to the element E21 that the user has gazed at isdecided on.

According to this embodiment, line-of-sight detection is performed byusing a so-called glasses-like head-mounted terminal 60. Thus, aline-of-sight position of a user can be accurately detected regardlessof the orientation of the face or body of the user. That is, the usercan naturally cast the eyes. Even if an angle of the head (theorientation of the face) of the user changes, line-of-sight detectioncan be accurately performed regardless of the change in the angle of thehead.

Example of Application to Medical System

FIG. 13 is an external view illustrating an electronic apparatus system(an endoscope system 10) in the case of using an endoscope apparatus 100as each of the electronic apparatuses 20 illustrated in FIG. 1.

Configuration of Endoscope Main Body

An endoscope main body 110 includes a handheld operation section 102 andan insertion section 104 that is connected to the handheld operationsection 102. An operator operates the handheld operation section 102while grasping handheld operation section 102 and inserts the insertionsection 104 into a body of a subject to perform observation. Theinsertion section 104 is constituted by a flexible portion 112, abending portion 114, and a distal end portion 116, which are arranged inthis order from the handheld operation section 102 side. The distal endportion 116 is provided with an imaging optical system 130 (see FIG.14), an illumination unit 123, a water supply nozzle 124, a forceps port126, and so forth.

At the time of observation or treatment, an operation of an operationunit 208 (see FIG. 14) enables either or both of visible light andinfrared light to be emitted from illumination lenses 123A and 123B ofthe illumination unit 123. In addition, an operation of the operationunit 208 enables washing water to be ejected from the water supplynozzle 124, so that a lens 132 of the imaging optical system 130 and theillumination lenses 123A and 123B can be washed. The forceps port 126allows a treatment tool, which is used to extirpate or remove a tumorand which is not illustrated, to be inserted therethrough, andaccordingly necessary treatment can be given to a subject by moving thetreatment tool forward or backward as appropriate.

As illustrated in FIG. 14, the lens 132 of the imaging optical system130 is disposed on a distal end surface of the distal end portion 116.An imaging device 134 of a complementary metal-oxide semiconductor(CMOS) type, a driving circuit 136, and an analog front end (AFE) 138are disposed behind the lens 132, so as to output an image signal. Inthis embodiment, a description will be given of a case where the imagingdevice 134 is a CMOS type imaging device, but the imaging device 134 maybe of a charge coupled device (CCD) type.

An observation image that has been taken through the lens 132 and soforth is formed on a light reception surface of the imaging device 134,converted into an electric signal, output to an endoscope processor 200through a signal cable that is not illustrated, and converted into avideo signal. Accordingly, the observation image is displayed on amonitor 400, which is connected to the endoscope processor 200.

Display of Observation Image

FIG. 15 is a diagram illustrating a state where the insertion section104 of the endoscope apparatus 100 is inserted into a subject, andillustrating a state where an observation image is obtained via theimaging optical system 130. In FIG. 15, a reference symbol IA denotes animaging range and a reference symbol tm denotes a tumor (a black raisedportion in FIG. 15).

As illustrated in FIGS. 13 to 15, the illumination lens 123A (forvisible light) and the illumination lens 123B (for infrared light) ofthe illumination unit 123 are disposed next to the lens 132 on thedistal end surface of the distal end portion 116. An emission end of alight guide 170, which will be described below, is disposed behind theillumination lenses 123A and 123B. The light guide 170 is insertedthrough the insertion section 104, the handheld operation section 102,and a universal cable 106, and an incidence end of the light guide 170is located in a light guide connector 108.

Configuration of Light Source Device

As illustrated in FIG. 14, a light source device 300 is constituted by alight source 310, a diaphragm 330, a condenser lens 340, and so forth,and causes illumination light to enter the light guide 170. The lightsource 310 includes a visible light source 310A and an infrared lightsource 310B and is capable of emitting one or both of visible light andinfrared light. Accordingly, with the light guide connector 108 (seeFIG. 13) being connected to the light source device 300, illuminationlight emitted by the light source device 300 is transmitted to theillumination lenses 123A and 123B through the light guide 170 and isapplied to an observation range from the illumination lenses 123A and123B.

Configuration of Endoscope Processor

Next, the configuration of the endoscope processor 200 will be describedwith reference to FIG. 14. In the endoscope processor 200, an imageinput controller 202 receives an image signal output from the endoscopeapparatus 100, an image processing unit 204 performs necessary imageprocessing thereon, and a video output unit 206 outputs a resultingimage signal. Accordingly, an observation image is displayed on themonitor 400. These processes are performed under control of a centralprocessing unit (CPU) 210. The image processing unit 204 performs, inaddition to image processing such as white balance adjustment, switchingor superimposition of an image displayed on the monitor 400, electroniczoom processing, display and switching of an image according to anoperation mode, or extraction of a specific component (for example, abrightness signal) from an image signal.

The endoscope processor 200 includes the operation unit 208. Theoperation unit 208 includes an operation mode setting/selecting switch,a water supply instruction button, or the like (not illustrated), and iscapable of performing an operation of emitting visible light or infraredlight.

Others

A description has been given of a case where the target apparatus to beoperated is the endoscope apparatus 100, which is a medical apparatus,with reference to FIG. 13. However, the target apparatus to be operatedin the present invention is not necessarily a medical apparatus. Theembodiments are applicable to an operation of a mobile apparatus, suchas a smartphone, a tablet terminal, or a game machine, as well as amedical apparatus, and to an operation of an apparatus under a situationwhere it is impossible to freely use both hands while operating anapparatus, for example, driving a car, or a situation where thearrangement of an apparatus is restricted. The target apparatus to beoperated is not limited to a single apparatus. One of a plurality ofapparatuses may be selected by using a line of sight and a motion of theneck. In the present invention, a gesture may be performed by movingonly the neck without moving the head, depending on the target apparatusto be operated, the usage situation of the apparatus, the type orimportance of an operation, or the like. A user as a target ofline-of-sight detection and a user as a target of gesture detection maybe different from each other.

Obviously, the present invention is not limited to the above-describedembodiments, and various changes can be made without deviating from thegist of the present invention.

REFERENCE SIGNS LIST

-   10 electronic apparatus system-   20 electronic apparatus-   22 database (an example of a voice recording unit)-   30 apparatus operation device-   31 display unit-   32 line-of-sight detecting camera-   33 neck-mounted terminal-   34 arithmetic unit-   38 communication unit-   39 storage unit (an example of a voice recording unit)-   40 CPU-   41 display control unit-   42 line-of-sight detecting unit-   43 action recognizing unit-   44 determining unit-   45 voice recognizing unit-   46 apparatus control unit-   52 neck motion sensor-   54 vibration sensor-   60 head-mounted terminal-   61 visual field camera-   62 line-of-sight detecting camera

What is claimed is:
 1. An apparatus operation device comprising: aline-of-sight detecting unit that detects a line of sight of a user; aneck-mounted terminal that is mounted around a neck of the user anddetects a motion of the neck of the user; a determining unit thatdetermines, based on the line of sight that has been detected and themotion of the neck that has been detected, at least one of a targetapparatus to be operated or operation details for the target apparatus;and a control unit that controls the target apparatus in accordance withthe determination by the determining unit.
 2. The apparatus operationdevice according to claim 1, wherein the determining unit determines,based on the line of sight that has been detected, at least one of thetarget apparatus or the operation details, and confirms thedetermination, based on the motion of the neck that has been detected.3. The apparatus operation device according to claim 1, wherein theline-of-sight detecting unit detects a movement of the line of sight,and the determining unit determines, based on the movement of the lineof sight that has been detected, at least one of the target apparatus orthe operation details.
 4. The apparatus operation device according toclaim 2, wherein the line-of-sight detecting unit detects a movement ofthe line of sight, and the determining unit determines, based on themovement of the line of sight that has been detected, at least one ofthe target apparatus or the operation details.
 5. The apparatusoperation device according to claim 1, wherein the operation detailsinclude a function that the control unit causes the target apparatus toexecute, and an execution condition for the function.
 6. The apparatusoperation device according to claim 2, wherein the operation detailsinclude a function that the control unit causes the target apparatus toexecute, and an execution condition for the function.
 7. The apparatusoperation device according to claim 3, wherein the operation detailsinclude a function that the control unit causes the target apparatus toexecute, and an execution condition for the function.
 8. The apparatusoperation device according to claim 4, wherein the operation detailsinclude a function that the control unit causes the target apparatus toexecute, and an execution condition for the function.
 9. The apparatusoperation device according to claim 1, comprising: a voice recognizingunit that recognizes a voice of the user by using a vibration of athroat of the user, the vibration being detected by the neck-mountedterminal.
 10. The apparatus operation device according to claim 2,comprising: a voice recognizing unit that recognizes a voice of the userby using a vibration of a throat of the user, the vibration beingdetected by the neck-mounted terminal.
 11. The apparatus operationdevice according to claim 3, comprising: a voice recognizing unit thatrecognizes a voice of the user by using a vibration of a throat of theuser, the vibration being detected by the neck-mounted terminal.
 12. Theapparatus operation device according to claim 4, comprising: a voicerecognizing unit that recognizes a voice of the user by using avibration of a throat of the user, the vibration being detected by theneck-mounted terminal.
 13. The apparatus operation device according toclaim 5, comprising: a voice recognizing unit that recognizes a voice ofthe user by using a vibration of a throat of the user, the vibrationbeing detected by the neck-mounted terminal.
 14. The apparatus operationdevice according to claim 6, comprising: a voice recognizing unit thatrecognizes a voice of the user by using a vibration of a throat of theuser, the vibration being detected by the neck-mounted terminal.
 15. Theapparatus operation device according to claim 9, comprising: a voicerecording unit that records the voice that has been recognized.
 16. Theapparatus operation device according to claim 1, wherein theline-of-sight detecting unit suspends detection of the line of sightupon detection of the motion of the neck being started by theneck-mounted terminal, and starts the detection of the line of sightupon the detection of the motion of the neck being finished by theneck-mounted terminal.
 17. The apparatus operation device according toclaim 1, wherein the determining unit invalidates detection of the lineof sight if the motion of the neck that has been detected is larger thana threshold.
 18. An apparatus operation method for an apparatusoperation device comprising a line-of-sight detecting unit that detectsa line of sight of a user and a neck-mounted terminal that is mountedaround a neck of the user and detects a motion of the neck of the user,the apparatus operation method comprising: a determination step ofdetermining, based on the line of sight that has been detected and themotion of the neck that has been detected, at least one of a targetapparatus to be operated or operation details for the target apparatus;and a control step of controlling the target apparatus in accordancewith the determination in the determination step.
 19. An electronicapparatus system comprising: the apparatus operation device according toclaim 1; and an electronic apparatus which is the target apparatus. 20.The electronic apparatus system according to claim 19, wherein theelectronic apparatus is a medical apparatus.